Imaging device and image processing apparatus

ABSTRACT

An imaging device includes an imager, with which an optical image of an object scene is repetitively captured. A first movement detector detects, as to each of a plurality of object scene images according to a time series output from the imager, a movement of a first feature point between the object scene image and the object scene images immediately before, and a clipper performs clipping processing on each of the plurality of object scene images on the basis of the detection result. When a still image recording operation is performed, a CPU changes an exposure time of the imager in such a direction as to shorten the time, and a second movement detector detects a movement of a second feature point between the object scene image immediately after the recording operation (reference object scene image) and the three object scene images being successive thereto out of a plurality of object scene images, and an adder adds the respective three object scene images to the reference object scene image while displacing the same on the basis of the detection result.

CROSS REFERENCE OF RELATED APPLICATION

The disclosure of Japanese Patent Application Nos. 2007-223105,2007-239282 and 2007-239283 are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging device and an imageprocessing apparatus. More specifically, the present invention relatesto an imaging device which electronically performs a camera shakecorrection, and an image processing apparatus which performs imageprocessing, such as a camera shake correction, a noise reduction, etc.on a plurality of object scene images according to a time series thatare captured by the image sensor.

2. Description of the Related Art

(I) In relation to imaging devices of such a kind, in one related art, astill image is produced by displacing a plurality of images beingcontinuous according to a time series and adding them to each other. Inanother related art, a motion image is produced by displacing andclipping a part out of each of the plurality of object scene imagesbeing continuous according to a time series.

In general, in an actual imaging device, such as a digital camera, etc.,it is required to shoot both of the motion images and the still imageswith high image quality. However, even if the related arts are merelycombined, it is difficult to shoot both of the motion images and stillimages with a proper exposure amount by accurately using two kinds ofcamera shake correction for motion image and still image.

(II) With respect to the image processing apparatus of such a kind, inone related art, by displacing and cutting a part from each of aplurality of object scene images being continuous according to a timeseries, a motion image with small movement among the object scene imagesdue to a camera shake (movement component in a time axis direction) isproduced. In another related art, a data for correction to correct anFPN (Fixed Pattern Noise) of the image sensor is stored in a memory, anda correction processing unit performs on the object scene image outputfrom the image sensor FPN correction processing on the basis of the datafor correction. In a still another related art, by displacing aplurality of object scene images being continuous according to a timeseries and making an addition to each other, it is possible to suppressa random noise of each object scene image (3DDNR: Three-DimensionalDigital Noise Reduction).

Generally, in an actual image processing apparatus, such as a digitalmovie camera, etc., a movement due to a hand shake, a random noise andan FPN are required to be suppressed. However, merely combining therelated arts with each other requires more memory and increases powerconsumption.

(III) Out of the image processing apparatuses of this kind, with respectto the one performing a 3DDNR, in one related art, by displacing aplurality of object scene images being continuous according to a timeseries and performing a weighted addition on them, a random noise ofeach object scene image is suppressed. In making the weighted addition,a coefficient to be multiplied by each object scene image is changeddepending on the magnitude of the movement among the object sceneimages. The magnitude of the movement among the object scene images isdetermined in unit of the object scene image (frame or field). Inanother related art, the magnitude of the movement is determined in unitof the pixel, and the 3DDNR is performed on only the part with smallmovement within each object scene image.

However, in the former related art, in a case that an object (hit ball,etc.) which is smaller in size and quickly moves is included in theobject scene, the image at the object portion is blurred due to the3DDNR. On the other hand, in the latter related art, the 3DDNR isperformed only on the part with small movement, and therefore, the blurat the portion with large movement is avoided, but the dimension of themovement is determined by the unit of pixels, resulting in an enormousamount of throughputs.

SUMMARY OF THE INVENTION

The present invention employs following features in order to solve theabove-described problems. It should be noted that reference numeralsinside the parentheses and the supplements show one example of acorresponding relationship with the embodiments described later for easyunderstanding of the present invention, and do not limit the presentinvention.

A first invention is an imaging device, comprising: an imager forrepetitively capturing an optical image of an object scene; a firstmovement detector for detecting, as to each of a plurality of objectscene images according to a time series output from the imager, amovement of a first feature point between the object scene image and oneor plurality of first proximate object scene images temporally proximateto the object scene image; a clipper for performing clipping processingon each of the plurality of object scene images on the basis of thedetection result by the first detector; a changer for changing an amountof the exposure to the imager in a such a direction as to reduce theamount in response to a still image recording operation; a secondmovement detector for detecting a movement of a second feature pointbetween a reference object scene image and one or plurality of secondproximate object scene images temporally proximate to the referenceobject scene image out of a plurality of object scene images accordingto a time series that is output from the imager after execution of thechanging processing by the changer; and a first adder for adding each ofsaid one or a plurality of second proximate object scene images to saidreference object scene images while displacing the same on the basis ofthe detection result by the second movement detector.

In an imaging device (10) of the first invention, an optical image of anobject scene is repetitively captured by an imager (14). A firstmovement detector (26) detects, as to each of a plurality of objectscene images according to a time series output from the imager, amovement of a first feature point between the object scene image and oneor plurality of first proximate object scene images temporally proximateto the object scene image (upper part in FIG. 3), and a clipper (22)performs clipping processing on each of the plurality of object sceneimages on the basis of the detection result by the first detector. Thus,it is possible to obtain an object scene image for motion image in whicha movement in a time axis direction (that is, movement among the frames)due to a hand shake is suppressed.

When a still image recording operation is performed, a changer (30)changes an amount of the exposure to the imager in such a direction asto reduce the amount (S3), a second movement detector (28) detects amovement of a second feature point between a reference object sceneimage and one or plurality of second proximate object scene imagestemporally proximate to the reference object scene image out of aplurality of object scene images according to a time series that isoutput from the imager after execution of the changing processing by thechanger (lower part in FIG. 3), and a first adder (32) displaces each ofthe one or plurality of second proximate object scene images on thebasis of the detection result by the second movement detector and addingthe same to the reference object scene image. Thus, it is possible toobtain an object scene image for still image in which a movement amongthe frames due to the hand shake is suppressed and an excessive exposuredue to the addition is reduced.

Here, the reduction in the amount of the exposure is realized bychanging an exposure time in such a direction as to shorten the time,and changing the aperture in such a direction as to close the aperture.In either case, the movement among the frames due to the hand shake isreduced, but in the former case, especially, if the exposure time isshortened to 1/N times, the movement among the frames becomes 1/N times.On the contrary thereto, in the latter case, the effect of the reductionis obtained since the movement component has a property as a randomnoise, but the movement component is not a complete random noise, andtherefore, the effect of the reduction in the latter case falls short ofthat in the former case.

A second invention is an image processing apparatus for performing imageprocessing on a plurality of object scene images according to a timeseries that are output from an imager for repetitively capturing anoptical image of an object scene, comprising: an FPN corrector forperforming FPN correction processing on the plurality of object sceneimages; a movement detector for detecting a movement of a feature pointamong the plurality of object scene images; a first clipper forperforming clipping processing on each of the plurality of object sceneimages at a position based on the detection result by the movementdetector; an adder for performing adding processing of adding to each ofthe plurality of object scene images after the first clipper, one orplurality of object scene images temporally proximate to the objectscene image; a first determiner for repetitively determining whether ornot the movement detected by the movement detector is above a thresholdvalue; and a first controller for making the FPN corrector invalid whenthe determination result by the first determiner is affirmative andmaking the FPN corrector valid when the determination result by thefirst determiner is negative.

In the second invention, an optical image of an object scene isrepetitively captured by an imager (114), and a plurality of objectscene images according to a time sequence that are output from theimager are applied to the image processing apparatus (100, 100A).

In the image processing apparatus, an FPN corrector (118 a) performs FPNcorrection processing on the plurality of object scene images, and amovement detector (122) detects a movement of a feature point among theplurality of object scene images. A first clipper (126) performsclipping processing at a position based on the detection result by themovement detector on each of the plurality of object scene images. Thatis, the clipping position of the first clipper moves according to themovement of the feature point. Thus, by making the clipping positionfollow the movement of the feature point, it is possible to reduce themovement among the plurality of object scene images.

Then, an adding processing of adding to each of the plurality of objectscene images after the first clipper, one or plurality of object sceneimages temporally proximate to the object scene image is performed by anadder (128, 128A). Thus, it is possible to reduce the random noiseincluded in each of the plurality of object scene images.

On the other hand, a first determiner (S105) repetitively determineswhether or not the movement detected by the movement detector is above athreshold value. A first controller (S107, S109) makes the FPN correctorinvalid when the determination result by the first determiner isaffirmative and making the FPN corrector valid when the determinationresult by the first determiner is negative.

Accordingly, while the movement is above the threshold value, the FPNcorrector is made invalid, capable of reducing power consumption. Thus,even if the FPN corrector is made invalid, the FPN is suppressed by theclipping processing by the first clipper and the adding processing bythe adder at the back thereof. The reason why is because by moving theclipping position, the FPN has a property as a random noise as a result,and can be reduced by the adder.

A third invention is an image processing apparatus for performing imageprocessing on a plurality of object scene images according to a timeseries that are output from an imager for repetitively capturing anoptical image of an object scene, comprising: a movement detector fordetecting a movement of a feature point among the plurality of objectscene images; a first clipper for performing clipping processing on eachof the plurality of object scene images at a position based on thedetection result by the movement detector; and an adder for performingadding processing of adding to each of the plurality of object sceneimages after the first clipper, one or plurality of object scene imagestemporally proximate to the object scene image, wherein the adderincludes a divider for dividing a pair of object scene images to beadded with each other into a common partial image, and a weighted adderfor weighing the division result by the divider with a coefficient foreach common partial image and adding the results to each other.

In the third invention, an optical image of an object scene isrepetitively captured by an imager (114), and a plurality of objectscene images according to a time sequence that are output from theimager are applied to the image processing apparatus (100, 100A).

In the image processing apparatus, a movement detector (122) detects amovement of a feature point among the plurality of object scene images,and a first clipper (126) performs clipping processing on each of theplurality of object scene images at a position based on the detectionresult by the movement detector. That is, the clipping position of thefirst clipper moves according to the movement of the feature point.Thus, by making the clipping position follow the movement of the featurepoint, it is possible to reduce the movement among the plurality ofobject scene images.

Each of the plurality of object scene images after the first clipper isapplied to the adder (128, 128A) so as to be subjected to addingprocessing of adding one or plurality of object scene images temporallyproximate to the object scene image. By thus adding the proximate objectscene images, it is possible to suppress the random noise included ineach of the plurality of object scene images.

In addition, by arranging an adder at the back of the first clipper, itis possible to also suppress the FPN. By moving the clipping position,the FPN has a property as a random noise, and such a noise component isalso suppressed by the adder.

Furthermore, in the adder, a divider (150 a, 150 b, 152 a, 152 b)divides a pair of the object scene images to be added with each otherinto a common partial image (B11-B66), and a weighted adder (154 a, 154b, 156) weighs the division result by the divider with a coefficient foreach common partial image and adds the results to each other. Bydetermining the coefficient for weighting for a common partial image, itis possible to avoid a blur at a part with the large movement.

Here, the common partial image may be a single pixel or blocks of mpixels×n pixels (m, n are integers equal to or more than one), but byproperly selecting the size of the common partial image (about severalpixels×several pixels), it is possible to suppress the throughput fordeciding the coefficient.

The above described objects and other objects, features, aspects andadvantages of the present invention will become more apparent from thefollowing detailed description of the present invention when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of a first embodimentof the present invention;

FIG. 2(A) is an illustrative view showing a movement detecting area formotion image applied to the first embodiment;

FIG. 2(B) is an illustrative view showing a movement detecting area forstill image applied to the first embodiment;

FIG. 3 is an illustrative view showing two kinds of image producingprocessing and two kinds of movement detecting processing along a timeaxis applied to the first embodiment;

FIG. 4(A) is an illustrative view showing a memory map of a memory formotion image at a certain time;

FIG. 4(B) is an illustrative view showing a memory map of a memory forstill image at the same time as FIG. 4(A);

FIG. 5 is an illustrative view showing a memory map of an SDRAM at thesame time as FIG. 4(A);

FIG. 6 is a timing chart showing an operation of the first embodiment;

FIG. 7 is a flowchart showing a part of an operation of a CPU applied tothe first embodiment;

FIG. 8 is a block diagram showing a configuration of a second embodimentof the present invention;

FIG. 9 is an illustrative view showing a part of a memory map of a DRAM;

FIG. 10 is an illustrative showing a part of a memory map of an internalmemory of a CPU 124;

FIG. 11 is an illustrative showing a part of a memory map of a flashmemory;

FIG. 12(A) is an illustrative view showing a clipping area of a firstclipping circuit;

FIG. 12(B) is an illustrative view showing a clipping area of a secondclipping circuit;

FIG. 13 is an illustrative view showing a correspondence betweendetection areas of a movement detection circuit and divided blocks of a3DDNR circuit;

FIG. 14 is a block diagram showing an example of the configuration ofthe 3DDNR circuit applied to the second embodiment;

FIG. 15 is an illustrative view showing a function utilized in weightedaddition processing by the 3DDNR circuit;

FIG. 16 is a timing chart showing an example of an operation of eachelement of the second embodiment;

FIG. 17 is a timing chart showing another example of the operation ofeach element of the second embodiment;

FIG. 18 is a flowchart showing a part of an operation of a CPU 124applied to a second embodiment;

FIG. 19 is a flowchart showing a part of an operation of a CPU 159applied to the second embodiment;

FIG. 20 is a block diagram showing a configuration of a modified exampleof the second embodiment;

FIG. 21 is a block diagram showing a configuration of a 3DDNR circuitapplied to the modified example;

FIG. 22 is a flowchart showing a part of an operation of the CPU 159applied to the modified example; and

FIG. 23 is a block diagram showing a configuration of the modifiedexample of the first embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment

Referring to FIG. 1, a digital camera 10 according to this embodimentincludes an image sensor 14. An optical image of an object scene passingthrough an optical lens 12 is irradiated onto an acceptance surface,that is, an imaging surface of the image sensor 14 on which electriccharges corresponding to the optical image of the object scene, that is,a raw image signal is generated by photoelectronic conversion.

When a real-time motion image of the object scene, that is, athrough-image is displayed on an LCD monitor 36, the CPU 30 instructsthe image sensor 14 to repetitively perform an exposure and a reading.The image sensor 14 repetitively executes an exposure over an exposuretime T and reading a raw image signal thus generated at a cycle of 1/60seconds, for example. A raw image signal corresponding to the opticalimage of the object scene is output from the image sensor 14.

The output raw image signal is subjected to preprocessing such as an A/Dconversion, a noise reduction, a clamp, a pixel defect correction, etc.by a signal preprocessing circuit 16. The original image data thusproduced is written to an original image area 60 (see FIG. 5) within anSDRAM 20 through a memory controller 18.

The original image area 60 includes a plurality of banks (60 here) eachhaving a capacity of one frame of an original image, and the memorycontroller 18 writes the original image data to these banks by oneframe. At this time, the CPU 30 applies a frame number (1, 2, . . . , k,. . . ) to each frame. After completion of writing to the last bank,similar writing processing is repeated from the first bank again. Thus,the original image area 60 always stores 60 frames in the immediatevicinity of the current frame.

The original image data generated by the signal preprocessing circuit 16is also applied to a movement detecting circuit for motion image 26 anda movement detecting circuit for still image 28. Each of the movementdetecting circuit for motion image 26 and the movement detecting circuitfor still image 28 detects a movement vector on the basis of the appliedimage data.

Here, in the movement detecting processing for motion image and themovement detecting processing for still image, the number of detectionareas assigned to the object scene is different between them. Morespecifically, the movement detecting circuit for motion image 26 assignsfive detection areas Em1-Em5 to the object scene (see FIG. 2(A)) whilethe movement detecting circuit for still image 28 assigns nine detectionareas Es1-Es9 to the object scene (see FIG. 2(B)).

Additionally, the movement detecting processing for motion image and themovement detecting processing for still image are different in a mannerof the movement vectors to be detected and a period during which thedetections are performed. That is, the movement detecting circuit formotion image 26 repetitively detects a movement of a feature pointbetween the current frame and the frame immediately before during athrough image/motion image recording.

On the contrary thereto, the movement detecting processing for stillimage is limited to a still image shooting period (see FIG. 6: describedlater) taking a time when the still image recording operation isperformed as a starting point. That is, assuming that a frameimmediately after the still image recording operation, that is, the“k”-th frame is taken as a reference frame, the movement detectingcircuit for still image 28 detects a movement of a feature point betweeneach of the three successive frames, that is, the “k+1”-th frame, the“k+2”-th frame and the “k+3”-th frame, and the reference frame. Thedetection result of the movement detecting circuit for motion image 26and the detection result of the movement detecting circuit for stillimage 28 are respectively written to a memory for motion image RI and amemory for still image R2 that are included in the CPU 30 (see FIG. 4(A)and FIG. 4(B)).

The original image data stored in the original image area 60 is read bythe memory controller 18, and applied to a camera shake correctioncircuit for motion image 22. On the other hand, the CPU 30 calculates aclipping position where the movement of the object scene image due to acamera shake is canceled out on the basis of the movement vector storedin the memory for motion image RI for each frame, and notifies theposition information indicating the calculation result to the camerashake correction circuit for motion image 22.

The camera shake correction circuit for motion image 22 executesclipping processing for clipping a part from the applied original imagedata on the basis of the notified clipping position information. Thus, amovement component in a time axis direction due to the hand shake isremoved from the original image data. The original image data on whichthe camera shake correction for motion image is performed is applied toa signal post-processing circuit 24 as image data for throughdisplay/motion image recording (hereinafter referred to as “motion imagedata”).

The signal post-processing circuit 24 performs post-processing such as acolor separation, a gamma correction, a YUV conversion, a pixel countconversion, etc. on the applied motion image data. The pixel countconversion processing here includes first number of pixels convertingprocessing for converting the number of pixels for motion image data tothe number of pixels for through display (that is, the number of pixelscorresponding to the resolution of the LCD monitor 36) and second numberof pixels converting processing for converting the number of pixels formotion image data into the number of pixels for motion image recording(640×480, for example). Accordingly, the signal post-processing circuit24 executes the first number of pixels converting processing and thesecond number of pixels converting processing in parallel (by timedivision, for example).

The motion image data on which the first number of pixels convertingprocessing is performed is written to a display area 66 (see FIG. 5)within an SDRAM 20 through the memory controller 18. An LCD driver 34reads the motion image data stored in the display area 66, and drivesthe LCD monitor 36 by the read motion image data. It should be notedthat the display area 66 includes two banks (not shown) each having acapacity of one frame of a through-image, and writing processing to onebank is performed in parallel with reading processing from the otherbank. Thus, a through-image of the object scene is displayed on themonitor screen.

The motion image data on which the second number of pixels convertingprocessing is performed is written to a motion image area 62 within theSDRAM 20 through the memory controller 18. The motion image area 62includes a plurality of banks (60, here) each having a capacity of oneframe of a motion image, and motion image data is written to these banksby one frame. After completion of writing to the last bank, similarwriting processing is repeated from the first bank again. Thus, themotion image area 62 always stores 60 frames in the immediate vicinityof the current frame.

The Y data out of the image data for through display output from thesignal post-processing circuit 24 is also applied to the CPU 30 for anexposure control. The CPU 30 adds up the applied Y data to generate aluminance evaluation value, and adjusts an exposure time T of the imagesensor 14 on the basis of the generated luminance evaluation value.

When a motion image record starting operation is performed with a keyinput device 44 thereafter, that is, during a through display, the CPU30 issues a compression instruction to an MPEG codec 38 a and opens anMPEG file as a writing destination of the compressed motion image withinthe recording medium 42. The MPEG codec 38 a reads the motion image databy a predetermined amount (15 frames, for example) from the motion imagearea 62 within the SDRAM 20 through the memory controller 18, andcompresses the read motion image data in an MPEG system. The compressedmotion image data output from the MPEG codec 38 a is written to acompression area 68 (see FIG. 5) within the SDRAM 20 through the memorycontroller 18.

The CPU 30 then reads the compressed motion image data stored in thecompression area 68 by the predetermined amount, and writes the readcompressed image data in an opened MPEG file through an I/F 40.

When a still image record starting operation is performed with the keyinput device 44 thereafter, that is, during the motion image recording,the destination for supplying the image data read from the originalimage area 60 is changed from the camera shake correction circuit formotion image 22 to the signal post-processing circuit 24. However, thechanged state is released after the still image shooting period (seeFIG. 6: described later) elapses.

Accordingly, assuming that the frame number immediately after the stillimage recording operation is “k”, image data at the k-th to the (k+3)-thframes are directly applied to the signal post-processing circuit 24 asimage data for still image recording (hereinafter referred to as “stillimage data”) without passing through the camera shake correction circuitfor motion image 22.

The image data for still image recording is also subjected to a similarpost-processing by the post-processing circuit 24, but the pixel countconversion processing here is third number of pixels convertingprocessing for converting the number of pixels of the image data intothe number of pixels for still image recording (1600×1200, for example).Thus, during the still image shooting period, in addition to the firstnumber of pixels converting processing and the second number of pixelsconverting processing with respect to the motion image data, the thirdnumber of pixels converting processing with respect to the still imagedata is executed in parallel.

The image data for still image recording output from the signalpost-processing circuit 24 is written to a still image area 64 withinthe SDRAM 20 through the memory controller 18. The still image area 64includes four banks each having a capacity of one frame of the stillimage, and in each bank, one frame of image data is written. Thus, thestill image area 64 stores four frames of image data immediately afterthe still image recording operation.

On the other hand, the CPU 30 changes the exposure time T of the imagesensor 14 to “T/4” in response to the still image recording operation,and then issues a correction instruction to a camera shake correctioncircuit for still image 32.

The camera shake correction circuit for still image 32 first calculateseach of the amount of displacements (Δ1, Δ2 and Δ3) of each of the(k+1)-th frame, the (k+2)-th frame, and the (k+3)-th frame with respectto the k-th frame on the basis of the movement vector stored in thememory for still image R2. Next, the still image data at the k-th to the(k+3)-th frames are read from the still image area 64 within the SDRAM20 through the memory controller 18. Next, the read still image data atthe (k+1)-th to the (k+3)-th frames are subjected to displacingprocessing on the basis of the calculated displacement information.Then, the still image data at the (k+1)-th to the (k+3)-th frames afterthe displacing processing is successively added to the read image dataat the k-th frame.

By the addition processing, it is possible to obtain the still imagedata with less blurring of image due to the hand shake and appropriatebrightness. It should be noted that in the addition, a coefficient(α0-α3) may be multiplied by the still image data of each frame(weighted addition).

The result of the aforementioned addition, that is, the still image dataoutput from the camera shake correction circuit for still image 32 iswritten to an addition area 70 (see FIG. 5) within the SDRAM 20 throughthe memory controller 18. It should be noted that the post-processingmay be performed not on the still image data to be added as describedabove, but on the added still image data stored in the addition area 70.

The CPU 30 then issues a compression instruction to the JPEG codec 38 bwhile opening a JPEG file as a writing destination of the compressedstill image within the recording medium 42. The JPEG codec 38 b readsthe still image data from the addition area 70 within the SDRAM 20through the memory controller 18, and compresses the read still imagedata in a JPEG system. The compressed still image data output from theJPEG codec 38 b is written to the compression area 68 (see FIG. 5)within the SDRAM 20 through the memory controller 18.

The CPU 30 then reads the compressed still image data stored in thecompression area 68, and writes the read compressed still image data tothe opened JPEG file through the I/F 40. It should be noted that such astill image recording can be performed not only during the motion imagerecording but also during the through image displaying.

When a motion image record stopping operation is performed by the keyinput device 44 thereafter, that is, during the motion image recording,the CPU 30 issues a stop instruction to the MPEG codec 38 a. Aftercompletion of writing the compressed image data, the MPEG file isclosed.

The operation of the digital camera 10 as described above is accordingto a timing chart shown in FIG. 6. Referring to FIG. 6, the exposure bythe image sensor 14 is performed at a fixed cycle (at a cycle of 1/60seconds) in both of the through image/motion image recording and thestill image recording. During the still image recording, that is, duringa little over three frames (k/60≦t≦(k+4)/60) from the time when thestill image recording operation is performed, the exposure time ischanged to ¼ times shorter than that in the through image/motion imagerecording.

The movement vector detection by the movement detecting circuit formotion image 26 is executed every time that an exposure by the imagesensor 14 is completed, and the clipping by the camera shake correctioncircuit for motion image 22 is performed every time that the movementvector detection by the movement detecting circuit for motion image 26is completed. At a time of the still image recording, the motion images“k”−“k+3”(see FIG. 3) are short of the amount of the exposure, andtherefore, writing to a new image to the motion image area 62 may beinterrupted and accordingly, the movement vector detection by themovement detecting circuit for motion image 26 may be suspended.

On the other hand, the movement vector detection for the movementdetecting circuit for still image 28 is performed every time that theexposure for still image recording, specifically, when each of thelatter three times out of the four times corresponding to the fourexposures of t=k/60, (k+1)/60, (k+2)/60 and (k+3)/60 is completed. Theaddition by the camera shake correction circuit for still image 32 isperformed at a time when a series of three movement vector detections bythe movement detecting circuit for still image 28 is completed, that is,when the movement vector detection corresponding to t=(k+3)/60 iscompleted.

In addition, as understood from FIG. 6, for producing the still imagefor recording, at least the exposure time (=T/4), the movement vectordetection time, and the addition time are required in addition to theperiod of three frames (=3/60 seconds), but each of these time periodsis so short in comparison with the three frames of period that the stillimage can be produced during the period of a little over three frames(the period of a little over three frames is called “still imageshooting period”). Thus, even if the writing of a new image to themotion image area 62 is interrupted during the still image shootingperiod, it has little effect on the image quality of the motion image.

However, by producing the motion image from the still image, the writingof a new image to the motion image area 62 may be continued even inrecording the still image. More specifically, with reference to FIG. 3,an image for recording taking the still image “k” as a reference isproduced, and three still images for recording taking the still images“k+1” to “k+3” as references are produced in a similar manner. Then, thepixel count conversion processing is performed on the four still imagesfor recording “k” to “k+3” to thereby obtain motion images for throughdisplay/recording “k” to “k+3”. Thus, even during the still imageshooting period, it is possible to maintain the image quality of themotion images. Furthermore, if the gain adjustment processing is alsoperformed in addition to the pixel count conversion processing, it ispossible to produce motion images with more appropriate brightness.

Or, also during the still image shooting period, writing of a novelimage to the motion image area 62 is continued, and gain-up processingequivalent to 4 times may be performed on the images stored in themotion image area 62. In this case, a gain adjustment circuit 46 shownby the dotted line in FIG. 1 is added to the digital camera 10.

Additionally, the movement detecting circuit for motion image 26 and themovement detecting circuit for still image 28 may be operated at leastfor one frame of period in parallel. This makes it possible to continueto freeze-display motion image frames corresponding to the still imagein the still image shooting period.

The processing of changing the exposure time T out of the processing bythe CPU 30 described above is according to an exposure time changingtask shown in FIG. 7. Additionally, the CPU 30 can execute in parallel aplurality of tasks including the task shown in FIG. 7 under the controlof the multitasking OS such as μITRON, etc.

Referring to FIG. 7, in a first step S1, it is determined whether or nota still image recording operation is performed, and if the determinationresult is “NO”, a standby condition holds. When a still image recordingoperation is accepted by the key input device 44, “YES” is determined inthe step S1, and the process shifts to a step S3. In the step S3, theexposure time T of the image sensor 14 is changed to ¼ times.

In a next step S5, it is determined whether or not a still imageshooting period (see FIG. 6) has elapsed since the still image recordingoperation was accepted, and if “NO”, a standby condition holds. If thedetermination result in the step S5 is “YES”, the process shifts to astep S7 to cancel the changed state. Thus, the exposure time of theimage sensor 14 is restored from “T/4” to “T”. Thereafter, the processreturns to the step S1.

As understood from the above description, in this embodiment, theoptical image of the object scene is repetitively captured by the imagesensor 14. The movement detecting circuit for motion image 26 detects,as to each of a plurality of object scene images output from the imagesensor 14, a movement of a feature point between the object scene imageand the object scene image immediately before (see upper part of FIG.3), and the camera shake correction circuit for motion image 22 performsclipping processing on each of the plurality of object scene images onthe basis of the detection result of the movement detecting circuit formotion image 26. Thus, it is possible to obtain an object scene imagefor through display/motion image recording with less movement in thetime axis direction (that is, the movement between the frames) due tothe hand sake.

When the still image recording operation is performed, the CPU 30changes the exposure time of the image sensor 14 in such a direction asto shorten the time (S3), the movement detecting circuit for still image28 detects the movement of the feature point between the object sceneimage (reference object scene image) directly after the still imagerecording operation and each of the three successive object scene imagesout of the plurality of object scene images (see lower part of FIG. 3),and the camera shake correction circuit for still image 32 adds in ordereach of the three object scene images to the reference object sceneimage while displacing each of the three object scene images on thebasis of the detection result of the movement detecting circuit forstill image 28.

Thus, it is possible to obtain an object scene image for still imagethat is reduced in movements among the frames due to the camera shakeand an excessive exposure by the addition.

Because two kinds of camera shake corrections for motion image and forstill image can properly be performed, it is possible to shoot both ofstill images and motion images with high image quality.

Furthermore, the movement detecting circuit for motion image 26 assignsthe five detection areas Em1-Em5 to the object scene (see FIG. 2(A))while the movement detecting circuit for still image 28 assigns the ninedetection areas Es1-Es9 more than the above described five to the objectscene (see FIG. 2(B)), and therefore, it is possible to perform amovement detection suitable for each of the motion image and the stillimage.

Additionally, the number of detection areas for motion image and thearrangement thereof are not restricted to those shown in FIG. 2(A), andthe number of detection areas for still image and the arrangementthereof are not restricted to those shown in FIG. 2(B). Here, it isdesirable that the detection areas for still image are arranged in amatrix with m rows and n columns (m is an integer equal to or more thanthree, and n is an integer equal to or more than three).

Moreover, in this embodiment, the movement detecting circuit for motionimage 26 detects the movement of the feature point between the currentframe and the frame immediately before, and the camera shake correctioncircuit for motion image 22 performs clipping on the basis of thedetection result. However, after the detecting processing by themovement detecting circuit for motion image 26, the CPU 30 predicts themovement of the feature point between the current frame and the frameimmediately after on the basis of the detection result, and the camerashake correction circuit for motion image 22 may perform clipping on thebasis of the prediction result.

Furthermore, assuming that a frame immediately after the still imagerecording operation, that is, the “k”-th frame is taken as a referenceframe, the movement detecting circuit for still image 28 detects amovement of a feature point between each of the three successive frames,that is, the “k+1”-th frame to the “k+3”-th frame, and the referenceframe, but the reference frame may be “k+1”-th frame without beingrestricted to the frame immediately after the still image recordingoperation. In this case, the frames to be compared with the referenceframe are not restricted to three frames succeeding to the referenceframe, but may be three frames before and after the reference frame,such as three frame of the “k”-th frame, the “k+2”-th frame, and the“k+3”-th frame.

In this embodiment, for producing a still image for recording, 4 framesare added, but generally, N frames (N is an integer equal to or morethan 2; preferably power of two) may merely be added. In this case, theexposure time is changed to 1/N times in response to a still imagerecording operation.

Furthermore, in this embodiment, in response to a still image recordingoperation, an exposure time is changed in such a direction as to shortenthe time, but in place of this or in addition this, the aperture of theimage sensor 14 may be changed in such a direction as to close theaperture. In brief, an excessive exposure due to the addition may bereduced by changing the amount of the exposure in such a direction as todecrease the same.

In the above description, the digital camera 10 is explained as oneexample, but the present invention can be applied to the imaging devicecapable of repetitively capturing an optical image of an object scene bythe image sensor, such as a digital still camera, a digital video(movie) camera, a mobile phone terminal with camera, etc.

By the way, in the first embodiment, a noise reduction is not especiallyexplained, but the adding processing executed in the camera shakecorrection circuit for still image 32 has an effect of suppressing therandom noise included in the object scene image. In a second embodimentto be explained next, before such an adding processing, by executingclipping processing as executed in the camera shake correction circuitfor motion image 22, it is possible to suppress an FPN as well as themovement due to a hand shake and the random noise.

Second Embodiment

With reference to FIG. 8, the digital movie camera 100 in thisembodiment includes an optical lens 112, an image sensor 114, a signalpreprocessing circuit 116, an FPN correction circuit 118 a, a linememory 118 b, a DRAM 120, a movement detection circuit 122, a CPU 124, afirst clipping circuit 126, a 3DDNR circuit 128, a signalpost-processing circuit 130, a second clipping circuit 132, a displaysystem circuit 134, a record system circuit 136, a flash memory 138 anda key input device 140.

Additionally, the image sensor 114 is preferably made up of a CMOS(Complementary Metal-Oxide Semiconductor), but may be made up of otherimage-pick-up devices, such as a CCD (Charge-Coupled Device), etc.

Out of these components, each of the FPN correction circuit 118 a, theline memory 118 b, the first clipping circuit 126 and the secondclipping circuit 132 is turned on or off by the CPU 124 on the basis ofthe presence or absence of FPN, the ON/OFF state of the camera shakecorrection mode, and the magnitude of a movement vector. The ON/OFFcontrol (see FIG. 18) is based on FPN correction and clipping on/offcontrolling program (182 d: see FIG. 11) stored in the flash memory 138.Furthermore, the 3DDNR circuit 128 is always placed at the ON state.

The configuration of the software of the digital movie camera 100 isexplained. FIG. 11 shows a part of a memory map of the flash memory 138.Referring to FIG. 11, the flash memory 138 is formed with an FPNinformation area 180, a program area 182, a mode information area 184and a corresponding information area 186.

The FPN information area 180 stores FPN information in relation to theimage sensor 114. The FPN information includes an FPN flag 180 a and asize, generating position and corrected value information 180 b. The FPNflag 180 a is a flag indicative of the presence or absence of an FPN,and the size, generating position and corrected value information 180 bis information describing the size of an FPN, the generating position ofthe FPN and the corrected value corresponding thereto. The size and thegenerating position of an FPN are common to a plurality of lines formingone frame, and therefore, the size, generating position and correctedvalue information 180 b are enough by one line. Here, if an FPN isabsent, the size, generating position and corrected value information180 b may be omitted.

The program area 182 stores a clipping position controlling program 182a, a line memory controlling program 182 b, a mode switch controllingprogram 182 c, a FPN correction and clipping on/off controlling program182 d, etc. The clipping position controlling program 182 a is a programfor calculating a clipping position on the basis of the detection resultby the movement detection circuit 122 (movement vector data stored inmovement vector areas 170-178 within a memory R integrated in the CPU124: see FIG. 10), and notifying the calculation result (clippingposition data stored in a clipping position area 179 within the memoryR) to the first clipping circuit 126.

The line memory controlling program 182 b is a program for reading asize, generating position and corrected value information 180 b from theFPN information area 180 (see FIG. 11) and writing it to the line memory118 b. The mode switch controlling program 182 c is for accepting a modeswitching operation by the key input device 140, and updating the modeinformation in the mode information area 184 according to the operationresult.

The FPN correction and clipping on/off controlling program 182 d is aprogram for performing an ON/OFF control of the FPN correction circuit118 a, the first clipping circuit 126 and the second clipping circuit132 on the basis of the presence or absence of FPN (FPN flag 180 a), anON/OFF state of the camera shake correction mode (camera shakecorrection mode flag 184 a: described later), and the magnitude of themovement vector (movement vector areas 170-178 for detection areasE11-E33: see FIG. 10).

The mode information area 184 stores the camera shake correction modeflag 184 a. The camera shake correction mode flag 184 a is a flag forindicating the ON/OFF state of the camera shake correction mode, andcontrolled by the mode switch controlling program 182 c.

The corresponding information area 186 stores information indicating acorrespondence (see FIG. 13) between the detection areas (E11-E33) ofthe movement detection circuit 122 and the blocks (B11-B66) as a unit ofthe weighted addition processing (described later) by the 3DDNR circuit128.

Furthermore, the above-described information and programs are written tothe flash memory 138 before being transferred. The camera shakecorrection mode flag 184 a is turned off in transferring, and is thenupdated in response to a mode switching operation by the key inputdevice 140. Furthermore, the CPU 124 can process these programs (182 a,182 b, . . . ) in parallels under the control of the multitasking OS,such as μITRON, etc.

Now, FIG. 18 shows a flowchart corresponding to the FPN correction andclipping on/off controlling program 182 d. With reference to FIG. 18,the CPU 124 first determines the presence or absence of an FPN on thebasis of the FPN flag 180 a (see FIG. 11) in a step S101. If thedetermination result in the step S101 is “YES”, that is, if “an FPN ispresent”, the process shifts to a step S103.

In the step S103, it is determined whether or not the camera shakecorrection mode is an ON state on the basis of the camera shakecorrection mode flag 184 a. If the determination result in the step S103is “YES”, that is, if “the camera shake correction is an ON state”, theprocess shifts to a step S105 while if the determination result is “NO”,that is, if “the camera shake correction is an OFF state”, the processshifts to a step S111.

In the step S105, it is determined whether or not the magnitude of themovement vector of the k-th frame with respect to the (k−1)-th frame(hereinafter referred to as “k:k−1”) is larger than a threshold value(Th1) on the basis of the movement vector data stored in the memory R.Here, the threshold value Th1 is determined in relation to the size ofthe FPN, and is preferably selected so as to take a value slightlylarger than the size of the FPN described in the size, generatingposition and corrected value information 180 b.

Referring to FIG. 10, the memory R includes the nine movement vectorareas 170-178 respectively corresponding to the nine detection areasE11-E33. The movement vector areas 170-178 respectively store dataindicating the movement vectors detected in the detection areas E11-E33.

If the magnitude of at least any one of the nine movement vectors“k:k−1” stored in the movement vector area 170-178 is above thethreshold value, “YES” is determined in the step S105 and the processshifts to a step S107. On the other hand, if all the magnitudes areequal to or smaller than the threshold value, “NO” is determined in thestep S105, and the process shifts to a step S109.

In the step S107, the FPN correction circuit 118 a is turned off, andthe first clipping circuit 126 is turned on while the second clippingcircuit 132 is turned off. In the step S109, the FPN correction circuit118 a is turned on, and the first clipping circuit 126 is turned onwhile the second clipping circuit 132 is turned off. In a step S111, theFPN correction circuit 118 a is turned on, and the first clippingcircuit 126 is turned off while the second clipping circuit 132 isturned on. Then, the process returns to the step S101.

If the determination result in the step S101 is “NO”, that is, if “anFPN is absent”, the process shifts to a step S113. In the step S113, itis determined whether or not the camera shake correction mode is an ONstate on the basis of the camera shake correction mode flag 184 asimilar to the step S103. If determination result in the step S113 is“YES”, the process shifts to a step S115 while if the determinationresult is “NO”, the process shifts to a step S117.

In the step S115, similar to the step S107, the FPN correction circuit118 a is turned off, and the first clipping circuit 126 is turned onwhile the second clipping circuit 132 is turned off. In the step S117,the FPN correction circuit 118 a is turned off, and the first clippingcircuit 126 is turned off while the second clipping circuit 132 isturned on. Then, the process returns to the step S101.

Accordingly, a state relating to FPN correction and clipping of thedigital movie camera 100 when an FPN is present is sifted among a statecorresponding to the period during which the step S107 is executed(hereinafter referred to as “S107 state”), a state corresponding to theperiod during which the step S109 is executed (S109 state), and a statecorresponding to the period during which the step S111 is executed (S111state).

On the other hand, a state relating to FPN correction and clipping ofthe digital movie camera 100 when an FPN is absent is shifted between astate corresponding to the period during which the step S115 is executed(S115 state) and a state corresponding to the period during which thestep S117 is executed (S117 state).

First, an operation in the “S107 state” is explained. Referring again toFIG. 8, an optical image of an object scene through the optical lens 112is irradiated onto the acceptance surface, that is, the imaging surfaceof the image sensor 114, on which electric charges, that is, a raw imagesignal corresponding to the optical image of the object scene isgenerated by photoelectronic conversion. The image sensor 114repetitively executes an exposure over an exposure time T and reading ofa raw image signal thus generated at a cycle of 1/60 seconds, forexample. The raw image signal corresponding to the optical image of theobject scene is output from the image sensor 114. The output raw imagesignal is subjected to preprocessing such as an A/D conversion, a clamp,etc. by a signal preprocessing circuit 116, to thereby produce raw imagedata. The produced raw image data is applied to the FPN correctioncircuit 118 a.

At this time, the FPN correction circuit 118 a is in a suspended state,and the writing processing (described later) to the line memory 118 b bythe CPU 124 is also not executed. The applied raw image data is writtento a first raw image area 160 (FIG. 9) of a DRAM 120 without passingthrough the FPN correction circuit 118 a.

The raw image data produced by the signal preprocessing circuit 116 isalso applied to the movement detection circuit 122. The movementdetection circuit 122 detects a movement vector on the basis of theapplied raw image data. Specifically, the nine detection areas E11-E33are assigned to the object scene (see FIG. 13), and the movement of thefeature point, that is, the movement vector is repetitively detectedbetween the current frame (k) and the frame (k−1) immediately before foreach detection area. The detection result of the movement detectioncircuit 122, that is, nine “k:k−1” respectively corresponding to thenine detection areas E11-E33 are written to the memory R integrated inthe CPU 124 (see FIG. 10).

The CPU 124 calculates for each frame a clipping position so as tocancel out the movement of the object scene image due to the hand shakeon the basis of the movement vector stored in the memory R, and notifiesthe clipping position information indicating the calculation result tothe first clipping circuit 126.

The raw image data stored in the first raw image area 160 is thenapplied to the first clipping circuit 126. The clipping area E1 of thefirst clipping circuit 126 is moveable (see FIG. 12(A)), and the firstclipping circuit 126 performs clipping processing of clipping a partcorresponding to the clipping area E1 from each frame on the applied rawimage data while moving the clipping area E1 on the basis of theclipping position information notified from the CPU 124. By the firstclipping processing, the movement component in a time axis directionincluded in the raw image data due to a hand shake is suppressed.

The raw image data output from the first clipping circuit 126 is thenapplied to the 3DDNR circuit 128. At this time, raw image data beforeone frame stored in the second raw image area 162 is further applied tothe 3DDNR circuit 128 as reference data, and the CPU 124 notifies themovement vector stored in the memory R to the 3DDNR circuit 128. Thatis, the raw image data of the two frame at the k-th and at the (k−1)-thand the nine movement vectors “k:k−1” between these two frames are inputat a common timing to the 3DDNR circuit 128.

The 3DDNR circuit 128 performs weighted addition processing on the basisof the nine movement vectors “k:k−1” on the raw image data at the k-thframe and the raw image data at the (k−1)-th frame. FIG. 14, FIG. 15 andFIG. 19 explain in detail the 3DDNR circuit 128 and the weightedaddition processing executed thereby.

Referring to FIG. 14, the 3DDNR circuit 128 includes a buffer 150 a anda controller 150 b, a buffer 152 a and a controller 152 b, a pair ofmultiplying circuits 154 a and 154 b, an adding circuit 156, a buffer158 a and a controller 158 b, and a CPU 159. The raw image data from thefirst clipping circuit 126 is written to the buffer 150 a and the rawimage data from the DRAM 120 is written to the buffer 152 a.

When the k-th frame is stored in the buffer 150 a, and the (k−1)-thframe is stored in the buffer 152 a, the controller 150 b and thecontroller 152 b respectively divide the frame stored in the buffer 150a and the buffer 152 a into 36 blocks of 6×6 (B11-B66: see FIG. 13), andreads the 36 blocks in order in response to a block reading instructionrepetitively issued by the CPU 159.

The read pair of blocks, that is, the block Bij read from the buffer 150a and the block Bij read from the buffer 152 a are respectively input tothe multiplying circuit 154 a and the multiplying circuit 154 b. At thistime, the CPU 159 specifies one corresponding to the block Bij out ofthe nine movement vectors “k:k−1”, and determines a coefficient α on thebasis of the specified movement vector.

The coefficient “α” is applied to the multiplying circuit 154 a, and themultiplying circuit 154 a multiplies the input block Bij by the appliedcoefficient “α”. On the other hand, the coefficient “1−α” is applied tothe multiplying circuit 154 b, and the multiplying circuit 154 bmultiplies the input block Bij by the coefficient “1−α”.

The multiplying result of the multiplying circuit 154 a and themultiplying result of the multiplying circuit 154 b are added to eachother in the adding circuit 156. Then, the adding result by the addingcircuit 156, that is, the block Bij to which the weighted addition isperformed is written to the buffer 158 a.

When the last block at the k-th frame, that is, the block B 66 iswritten to the buffer 158 a, the CPU 159 issues a frame readinginstruction to the controller 158 b, and the controller 158 b outputs 36blocks stored in the buffer 158 a as one frame.

The aforementioned processing by the CPU 159 is according to a flowchartshown in FIG. 19. Referring to FIG. 19, in a first step S131, it isdetermined whether or not each of the buffer 150 a and buffer 152 astores a frame, and if the determination result is “NO”, a standby stateis held. If the determination result in the step S131 is “YES”, theprocess shifts to a step S133 to issue a block reading instruction toeach of the controller 150 b and the controller 152 b. Thereafter, theprocess proceeds to a step S135.

In the step S135, it is determined whether or not the movement between apair of the blocks read in response to the block reading instruction,that is, the block Bij at the k-th frame and the block Bij at the(k−1)-th frame is large on the basis of the movement vector informationnotified from the CPU 124.

More specifically, first, the detection area corresponding to the pairof blocks is specified on the basis of the corresponding information 186a-186 i stored in the corresponding information area 186. For example,if the pair of blocks is “B11”, because “B11” is described in thecorresponding information 186 a, the corresponding detection area isfound to be “E11”.

Next, the movement vector corresponding to the detection area specifiedas described above is selected out of the nine movement vectors “k:k−1”respectively corresponding to the detection areas E11-E33 notified fromthe CPU 124, and the magnitude of the selected “k:k−1” is compared witha threshold value (Th2). Then, if “|k:k−1|>Th2”, it is determined themovement is large while if “|k:k−1≦Th2”, it is determined that themovement is small.

If the determination result in the step S135 is “YES”, that is, if themovement is large, the coefficient α is regarded as a maximum value αmax(0.8, for example) in a step S137. If the determination result in thestep S135 is “NO”, that is, if the movement is small, the coefficient αis calculated in a step S139. The calculating processing is based on afunction or a table defining a relationship between the movement, thatis, |k:k−1| and the coefficient α. FIG. 15 shows one example of such afunction.

Referring to FIG. 15, the function corresponds to the line segmenthaving two points (0, 0.5) and (Th2, 0.8) at both ends, and if themovement is in the section above Th2, α=0.8. Generally, it is onlynecessary be the function in which when the movement is “0”, α becomes aminimum, as the movement increases, α is made large, and when themovement is “Th2”, a is made a maximum. In a case of utilizing a table,distributed values calculated from the function, such as (0, 0.5), (1,0.6), . . . , (Th2, 0.8) are registered.

Returning to FIG. 19, when a coefficient is determined in the step S137or S139, the process proceeds to a step S141. In the step S141, the “α”and “1−α” are respectively notified to the multiplying circuit 154 a and154 b. In a succeeding step S143, it is determined whether or not theblock read immediately before is the last block of the frame, and if“NO”, the process returns to the step S133. If “YES” in the step S143,the process shifts to a step S145 to issue a frame outputtinginstruction to the controller 158 b, and then, the process returns tothe step S131.

By such weighted addition processing, it is possible to suppress therandom noise included in the raw image data.

Referring again to FIG. 8, the raw image data output from the 3DDNRcircuit 128 is written to the second raw image area 162 (see FIG. 9) ofthe DRAM 120. The image data thus stored in the second raw image area162 is then applied to the 3DDNR circuit 128 as reference data.

The raw image data output from the 3DDNR circuit 128 is also applied tothe signal post-processing circuit 130. The signal post-processingcircuit 130 performs post-processing, such as a color separation, agamma correction, a YUV conversion, etc. on the applied raw image data.Thus, the raw image data is converted to the YUV image data.

The YUV image data thus obtained is written to a YUV area 164 (see FIG.9) within the DRAM 120. The YUV image data stored in the YUV area 164 isthen applied to the second clipping circuit 132. At this time, thesecond clipping circuit 132 is in a suspended state, and therefore, theapplied YUV image data is applied to each of the display system circuit134 and the record system circuit 136 without passing through the secondclipping circuit 132.

In the display system circuit 134, processing of driving an LCD monitor(not shown) with the applied YUV image data, etc. is executed. In therecord system circuit 136, processing of compressing the applied YUVimage data, recording the compressed image data in a recording medium(not shown), etc. are executed.

Next, an operation of the “S109 state” is explained. Referring again toFIG. 8, the image sensor 114 repetitively executes exposure and readingof the raw image signal thus generated similar to the “S107 state”. Theraw image signal output from the image sensor 114 is subjected topreprocessing in the signal preprocessing circuit 116, and the raw imagedata thus generated is applied to the FPN correction circuit 118 a inlines.

The CPU 124 reads the size, generating position and corrected valueinformation 180 b (see FIG. 11) from the flash memory 138, and writesthe same in the line memory 118 b. The writing processing is based onthe line memory writing controlling program 182 b (see FIG. 11). The FPNcorrection circuit 118 a performs an FPN correction on the applied rawimage data on the basis of the size, generating position and correctedvalue information 180 b stored in the line memory 118 b. Thus, the FPNincluded in the raw image data is suppressed.

The raw image data stored in the first raw image area 160 is thenapplied to the first clipping circuit 126 so as to be subjected tosimilar first clipping processing.

At this time, as in the “S107 state”, the raw image data before oneframe stored in the second raw image area 162 is applied to the 3DDNRcircuit 128 as reference data. The movement detection circuit 122performs similar movement vector detection, and notifies the detectionresult to the 3DDNR circuit 128 through the CPU 124. The 3DDNR circuit128 executes weighted addition processing on the applied raw image dataon the basis of the notified movement vector.

The raw image data output from the 3DDNR circuit 128 is written to thesecond raw image area 162 of the DRAM 120, and then applied to the 3DDNRcircuit 128 as reference data. The raw image data output from the 3DDNRcircuit 128 is also applied to the signal post-processing circuit 130 soas to be converted into YUV image data. The YUV image data is written tothe YUV area 164 within DRAM 120, and then applied to the secondclipping circuit 132.

At this time, as in the “S107 state”, the second clipping circuit 132 isin a suspended state, and the applied YUV image data is applied to eachof the display system circuit 134 and the record system circuit 136without passing through the second clipping circuit 132. In the displaysystem circuit 134, LCD driving processing, and etc. is performed, andin the record system circuit 136, compression processing, recordingprocessing, and etc. are executed.

Here, a state transition from the “step S107 state” to the “step S109state” is shown in a timing chart in FIG. 16. Referring to FIG. 16,while the magnitude of the movement vector is equal to or smaller thanthe threshold value, each of the first clipping processing, the FPNcorrection processing and the 3 DDNR processing is continued, and thesecond clipping processing is suspended. If the magnitude of themovement vector is above the threshold value, the FPN correctionprocessing is stopped. Each of the first clipping processing and 3DDNRprocessing is continued, and the second clipping processing remains tobe stopped.

Next, an operation of the “S111 state” is explained. The “S111 state” isa sate in which the first clipping circuit 126 is suspended and thesecond clipping circuit 132 is started in the “109 state”. Accordingly,in the “step S111 state”, the clipping position calculating andnotifying processing by the CPU 124 is not executed. Thus, the raw imagedata of the first raw image area 160 is applied to the 3DDNR circuit 128without passing through the first clipping circuit 126. On the otherhand, the YUV image data from the YUV area 164 is subjected to theclipping processing by the second clipping circuit 132, and then appliedto each of the display system circuit 134 and the record system circuit136.

The clipping area E2 of the second clipping circuit 132 is fixed (FIG.12(B)), and the second clipping circuit 132 performs on the applied YUVimage data clipping processing of clipping a part corresponding to theclipping area E2 from each frame. The change in the image size bystopping/restarting the first clipping processing is cancelled byexecuting the second clipping processing. Other than this, the operationis similar to that in the “S109 state”.

Here, a state transition from the “S107 state” to the “S111 state” isshown in a timing chart in FIG. 17. Referring to FIG. 17, while thecamera shake correction is turned on, each of the first clippingprocessing and the 3DDNR processing is continued, and each of the secondclipping processing and the FPN correction processing is suspended. Whenthe camera shake correction is turned off, the first clipping processingis stopped, and each of the FPN correction processing and the secondclipping processing is activated. The 3DDNR processing is continued.

Last, an operation in each of the S115 sate and the S117 state isbriefly explained. The “S115 state” is the same as the “S107 state”, andthe operations to be performed are common to each other.

The “S117 state” is a state in which the FPN correction circuit 118 a isfurther suspended in the “S111 state”. Accordingly, in the “step S117state”, the raw image data from the signal preprocessing circuit 116 iswritten to the first raw image area 160 of the DRAM 120 without passingthrough the FPN correction circuit 118 a. Other than this, the operationis the same as that in the “S111 state”.

As understood from the foregoing, in this embodiment, an optical imageof the object scene is repetitively captured by the image sensor 114,from which a plurality of object scene images according to a time seriesare output. The FPN correction circuit 118 a performs FPN correctionprocessing on the plurality of object scene images, and the movementdetection circuit 122 detects the movement of the feature point betweenthe plurality of object scene images.

The first clipping circuit 126 performs clipping processing on each ofthe plurality of object scene images on the basis of the detectionresult by the movement detector. That is, the clipping position by thefirst clipping circuit 126 moves in correspondence with the movement ofthe feature point. Thus, by making the clipping position follow themovement of the feature point, it is possible to suppress the movementamong the plurality of object scene images.

Then, adding processing of adding the object scene image immediatelybefore to each of the plurality of object scene images according to thetime series on which the clipping processing by the first clippingcircuit 126 is performed is performed by the 3DDNR circuit 128. Thus, itis possible to suppress the random noise included in each of theplurality of object scene images.

On the other hand, the CPU 124 repetitively determines whether or notthe movement detected by the movement detector is above the thresholdvalue (S105), and when the determination result is affirmative, the FPNcorrection circuit 118 a is made invalid (S107) while if thedetermination result is negative, the FPN correction circuit 118 a ismade valid (S109).

Accordingly, while the movement is above the threshold value, the FPNcorrection circuit 118 a is made invalid, and therefore, it is possibleto reduce power consumption. Thus, even if the FPN correction circuit118 a is made invalid, the FPN is suppressed by the clipping processingby the first clipping circuit 126 and the following adding processing bythe 3DDNR circuit 128. The reason why is because by the movement of theclipping position, the FPN has a property as a random noise as a result,and is suppressed in the 3DDNR circuit 128.

Furthermore, in this embodiment, in the 3DDNR circuit 128, each of thebuffer 150 a and the controller 150 b, and the buffer 152 a and thecontroller 152 b divides a pair of object scene images to be added witheach other into blocks B11-B66 (common partial image), and themultiplying circuits 154 a and 154 b and the adding circuit 156 assign aweight to the division result by the coefficient for each block and addthe weighed results. The coefficient (α) for weighing is determined foreach block by the CPU 159 on the basis of the movement vector notifiedfrom the movement detection circuit 122 via the CPU 124. Thus, bydeciding the coefficient for each common partial image, it is possibleto avoid a blur at a portion with large movements.

Then, the blocks B11-B66 may be a single pixel or m pixels×n pixels (mand n are integers equal to or more than one), but by appropriatelyselecting the size of the block (several pixels×several pixels, forexample), it is possible to reduce a throughput for deciding thecoefficient. Thus, it is possible to appropriately suppress the randomnoise with a low throughput.

Additionally, in the 3DDNR circuit 128 in this embodiment, the CPU 159decides the coefficient α on the basis of the movement vector notifiedfrom the movement detection circuit 122 via the CPU 124, butalternatively, a difference between a pair of block Bij and Bijrespectively output from the buffer 150 a and the buffer 152 a isevaluated, and on the basis of the difference, the coefficient may bedecided. Such modified example is shown in FIG. 20 and FIG. 21.

Referring to FIG. 20, a digital movie camera 100A is an example in whichin the digital movie camera 100 in FIG. 8, the notification of themovement vector form the CPU 124 to the CPU 159 is omitted, and the3DDNR circuit 128 is replaced with a 3DDNR circuit 128A.

Referring to FIG. 21, the 3DDNR circuit 128A is an example in which tothe 3DDNR circuit 128 shown in FIG. 14, a difference calculating circuit153 for calculating the difference between the blocks Bij and Bij outputfrom the buffers 150 a and 152 a and a coefficient calculating circuit155 for calculating a coefficient α on the basis of this calculationresult are added.

The calculation result by the difference calculating circuit 153 isapplied to each of the coefficient calculating circuit 155 and the CPU159. The coefficient calculating circuit 155 evaluates the coefficient αon the basis of the applied difference from a following Eq. (1).

α={difference}×βη  (1)

Here, β and η are parameters respectively corresponding to aninclination and an intercept of the straight line, and set by the CPU159.

The CPU 159 decides the parameters β and η in the aforementioned Eq. (1)on the basis of the applied difference, and sets the decision result tothe difference calculating circuit 153. FIG. 22 shows a flowchart of anoperation of the CPU 159 in this case.

The flowchart shown in FIG. 22 has steps S134 and S136 in place of thesteps S135-S141 in the flowchart in FIG. 19. In the step S134, theparameters β and η are decided on the basis of the difference from thedifference calculating circuit 153, and the decision result is set tothe coefficient calculating circuit 155 in the step S136. In response tothe coefficient setting processing, the coefficients α and “1−α” arerespectively output to the multiplying circuits 154 a and 154 b from thecoefficient calculating circuit 155.

It should be noted that the parameters β and η may be decidedirrespective of the difference, or may be fixed values. In such a case,the calculation result by the difference calculating circuit 153 is notrequired to be notified.

Thus, it is possible to reduce the random noise more properly.

Furthermore, in this embodiment (or the modified example), the number ofdetection areas of the movement detection circuit 122 is nine, and thesenine detection areas are arranged in a matrix manner of 3 lines by 3columns (see FIG. 13), but the number and the size are not restrictedthereto. For example, the four detection areas E11, E31, E13 and E33arranged at the four corners of the frame are omitted, and at theportion to which the detection area is not assigned, the coefficient αmay be fixed to a minimum value (0.5, for example). Here, in order tosuitably remove the movement components between the frames due the handshake in the first clipping circuit 126, and properly remove the randomnoise (including the movement components between the frames due the handshake) in the 3DDNR circuit 128, 128A, a matrix manner of m lines by ncolumns (m is an integer equal to or more than three, and n is aninteger equal to or more than three) is preferable.

Additionally, in this embodiment, there are spaces between therespective detection areas E11-E33, but these spaces may be eliminated.The detection areas adjacent to each other may be overlapped with eachother. The shape of the detection area may take a shape of a circle, apolygon, etc. without being restricted to a rectangle.

Furthermore, in this embodiment, the movement detection circuit 122detects a movement of the feature point between the current frame (k)and the frame immediately before (k−1), and the first clipping circuit126 performs a clipping on the basis of the detection result. However,after the detecting processing by the movement detection circuit 122,the CPU 124 predicts the movement between the current frame (k) and theframe immediately after (k+1), and the first clipping circuit 126 mayperform a clipping on the basis of the prediction result. In a case ofFIG. 20 embodiment, the 3DDNR circuit 128 may also use the predictionresult.

Moreover, in this embodiment, the FPN information is fixed, but the CPU124 may always detect an FPN of the image sensor 114 through the FPNcorrection circuit 118 a, and update the FPN information on the basis ofthe detection result, for example.

In the foregoing, the digital movie camera (100, 100A) is explained asone example, but the present invention can be applied to imageprocessing apparatuses capable of processing a plurality of object sceneimages according to a time series captured by the image sensor. Theimage processing apparatuses in this case includes not only ones havingthe image sensor as a component (digital still camera, mobile terminalwith camera, etc.) but also ones not having the image sensor as acomponent (personal computer, digital video reproducing device, etc.capable of capturing an object scene image taken by an external imagesensor, for example).

For example, in a case that the invention is applied to the digitalcamera 10 of the first embodiment, the 3DDNR circuit 128 shown in FIG. 8(or the 3DDNR circuit 128A in FIG. 20) is added to the digital camera 10as shown in FIG. 23. The 3DDNR circuit 128 performs on each of theplurality of object scene images according to the time series after theclipping processing by the camera shake correction circuit for motionimage 22 adding processing of displacing the object scene imageimmediately before the object image on the basis of the detection resultby the movement detecting circuit for motion image 26 and adding themduring the motion image shooting. Thus, it is possible to suppress therandom noise included in each of the plurality of object scene images.Furthermore, in place of adding the 3DDNR circuit 128, the camera shakecorrection circuit for still image 32 may be worked as a 3DDNR circuit128 during the motion image shooting period.

Furthermore, in FIG. 23 example, the FPN correction circuit 118 a andthe line memory 118 b are added to the digital camera 10, and the CPU 30repetitively determines whether or not the detection result of themovement detecting circuit for motion image 26 is above the thresholdvalue during the motion image shooting (corresponding to the step S105in FIG. 18), and if the determination result is affirmative, that is, ifthe detection result is above the threshold value, the FPN correctioncircuit 118 a is made invalid (corresponding to the step S107) while ifthe determination result is negative, that is, if the detection resultis not above the threshold value, the FPN correction circuit 118 a ismade valid (corresponding to the step S109).

In addition, the 3DDNR circuit 128 divides each of the pair of objectscene images to be added to each other into the common partial images(B11-B66), and by assigning a weight with a coefficient for each commonpartial image and adding the resultants to each other, it is possible toavoid a blur at a portion with a large movement.

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the spiritand scope of the present invention being limited only by the terms ofthe appended claims.

1. An imaging device, comprising: an imager for repetitively capturingan optical image of an object scene; a first movement detector fordetecting, as to each of a plurality of object scene images according toa time series output from said imager, a movement of a first featurepoint between said object scene image and one or plurality of firstproximate object scene images temporally proximate to said object sceneimage; a clipper for performing clipping processing on each of saidplurality of object scene images on the basis of the detection result bysaid first detector; a changer for changing an amount of the exposure tosaid imager in such a direction as to reduce the amount in response to astill image recording operation; a second movement detector fordetecting a movement of a second feature point between a referenceobject scene image and one or plurality of second proximate object sceneimages temporally proximate to said reference object scene image out ofa plurality of object scene images according to a time series that areoutput from said imager after execution of the changing processing bysaid changer; and a first adder for adding each of said one or pluralityof second proximate object scene images to said reference object sceneimage while displacing the same on the basis of the detection result bysaid second movement detector.
 2. An imaging device according to claim1, wherein said changer changes an amount of the exposure in such adirection as to reduce the amount by shortening an exposure time.
 3. Animaging device according to claim 1, wherein said changer changes to aninverse times of a total number of the reference object scene image andthe second proximate object scene images to be added with each other bysaid first adder.
 4. An imaging device according to claim 3, whereinsaid total number is a power of two.
 5. An imaging device according toclaim 1, wherein said changer cancels the changed state after saidimaging device completes the output of the reference object scene imageand the second proximate object scene images to be added by said firstadder.
 6. An imaging device according to claim 1, wherein said firstmovement detector assigns a first detection area of a first presetnumber to said object scene, and said second movement detector assigns asecond detection area of a second preset number being more in numberthan said first preset number to said object scene.
 7. An imaging deviceaccording to claim 6, wherein the clipping processing by said clipper isbased on at least one of the detection results of said first presetnumber by said first movement detector, and the adding processing bysaid first adder is based on at least one of the detection results ofsaid second preset number by said second movement detector.
 8. Animaging device according to claim 6, wherein said second preset numberis m×n (m is an integer equal to or more than three, and n is an integerequal to or more than three), and said detection area of the secondpreset number is arranged in a matrix with m rows and n columns.
 9. Animaging device according to claim 1, wherein said first movementdetector and said second movement detector operate in parallel with eachother.
 10. An imaging device according to claim 1, further comprising: amotion image recorder for performing motion image recording processingon said plurality of object scene images after the clipping processingby said clipper; and a still image recorder for performing still imagerecording processing on the reference object scene image to which saidone or plurality of second proximate object scene images have alreadybeen added by said first adder.
 11. An imaging device according to claim10, further comprising: a calculator for calculating a displacementamong said plurality of object scene images on the basis of at least anyone of the detection result from said first movement detector and saidsecond movement detector, wherein said calculator selectively executesfirst calculating processing for calculating a displacement between saidobject scene image and one or plurality of first proximate object sceneimages temporally proximate to said object scene image as to each ofsaid plurality of object scene images, and second calculating processingfor calculating a displacement between said reference object scene imageand one or plurality of second proximate object scene images temporallyproximate to said reference object scene image out of said plurality ofobject scene images depending on an operating state of said motion imagerecorder and said still image recorder, and said clipper and said firstadder perform processing with reference to the calculation result bysaid calculator.
 12. An imaging device according to claim 11, furthercomprising: a display for performing motion image displaying processingon said plurality of object scene images after the clipping processingby said clipper, wherein an output by said imager is branched to aplurality of systems including a first system directed to at least anyone of said display, said motion image recorder and said still imagerecorder, a second system directed to said first movement detector, anda third system directed to said second movement detector.
 13. An imagingdevice according to claim 12, wherein said first movement detector andsaid second movement detector operates in parallel with each other atleast for one frame of period.
 14. An imaging device according to claim13, further comprising a gain increaser for increasing a gain to a totalnumber of the reference object scene image and the second proximateobject scene images to be added with each other by said first adderduring which inputs to said motion image recorder are performed by saidmotion image recorder and said still image recorder in parallel witheach other.
 15. An imaging device according to claim 1, furthercomprising a second adder for adding to each of said plurality of objectscene images after said first clipper, one or plurality of object sceneimages temporally proximate to said object scene image.
 16. An imagingdevice according to claim 15, further comprising: a FPN corrector forperforming FPN correction processing on said plurality of object sceneimages; a determiner for repetitively determining whether or not themovement detected by said first movement detector is above a thresholdvalue; and a controller for making said FPN corrector invalid when thedetermination result by said determiner is affirmative, and making saidFPN corrector valid when the determination result by said determiner isnegative.
 17. An imaging device according to claim 15, wherein saidsecond adder includes a divider for dividing a pair of object sceneimage to be added with each other into common partial images, and aweighted adder for weighing the division result by said divider with acoefficient for each common partial image and adding the results to eachother.
 18. An imaging device, comprising: an imager repetitivelycapturing an optical image of an object scene; a movement detector fordetecting, out of a plurality of object scene images that is output fromsaid imager, a movement of a feature point between a reference objectscene image and one or plurality of proximate object scene imagestemporally proximate to said reference object scene image; and a adderfor adding each of said one or plurality of proximate object sceneimages to said reference object scene image while displacing the same onthe basis of the detection result by said movement detector, whereinsaid movement detector assigns a detection area arranged in a matrixwith m rows and n columns (m is an integer equal to or more than three,n is an integer equal to or more than three) to said object scene, andthe adding processing by said adder is based on at least one detectionresult out of the detection results corresponding to each element ofsaid m×n detection area.
 19. An image processing apparatus forperforming image processing on a plurality of object scene imagesaccording to a time series that is output from an imager forrepetitively capturing an optical image of an object scene, comprising:an FPN corrector for performing FPN correction processing on saidplurality of object scene images; a movement detector for detecting amovement of a feature point among said plurality of object scene images;a first clipper for performing clipping processing on each of saidplurality of object scene images at a position based on the detectionresult by said movement detector; an adder for performing addingprocessing of adding to each of said plurality of object scene imagesafter said first clipper, one or plurality of object scene imagestemporally proximate to said object scene image; a first determiner forrepetitively determining whether or not the movement detected by saidmovement detector is above a threshold value; and a first controller formaking said FPN corrector invalid when the determination result by saidfirst determiner is affirmative and making said FPN corrector valid whenthe determination result by said first determiner is negative.
 20. Animage processing apparatus according to claim 19, further comprising afirst memory to which FPN information inherent to said imager iswritten, wherein said FPN corrector executes FPN correction processingon the basis of the FPN information stored in said first memory.
 21. Animage processing apparatus according to claim 20, wherein said FPNinformation includes size information indicating a size of the FPN, andsaid threshold value is a value in relation to said size information.22. An image processing apparatus according to claim 21, wherein saidthreshold value is larger than the value indicated by said sizeinformation.
 23. An image processing apparatus according to claim 20,wherein said FPN information includes presence or absence informationindicating the presence or absence of an FPN, further comprising: asecond memory to which mode information indicating whether a camerashake correction mode is valid or invalid is written; and an updater forupdating the mode information stored in said second memory in responseto a mode switching operation, wherein said first determiner executesdetermining processing when the presence or absence information storedin said first memory indicates that an FPN is present, and the modeinformation stored in said second memory indicates that the camera shakecorrection is valid.
 24. An image processing apparatus according toclaim 23, wherein said first controller makes said first clipper validregardless of the determination result by said first determiner.
 25. Animage processing apparatus according to claim 23, further comprising asecond controller for making said FPN corrector valid and making saidfirst clipper invalid when the presence or absence information stored insaid first memory indicates that an FPN is present and the modeinformation stored in said second memory indicates that the camera shakecorrection mode is invalid.
 26. An image processing apparatus accordingto claim 25, further comprising a third controller for making said FPNcorrector invalid and making said first clipper valid when the presenceor absence information indicates that an FPN is absent and the modeinformation stored in said second memory indicates that the camera shakecorrection mode is valid.
 27. An image processing apparatus according toclaim 26, further comprising a fourth controller for making each of saidFPN corrector and said first clipper invalid when the presence orabsence information indicates that an FPN is absent and the modeinformation stored in said second memory indicates that the camera shakecorrection mode is invalid.
 28. An image processing apparatus accordingto claim 27, further comprising a second clipper for performing clippingprocessing on each of said plurality of object scene images at a fixedposition, wherein each of said first controller and said thirdcontroller further makes said second clipper invalid in response to thevalidating processing of said first clipper, and each of said secondcontroller and said fourth controller further makes said second clippervalid in response to the invalidating processing by said first clipper.29. An image processing apparatus according to claim 19, wherein saidadder includes a divider for dividing a pair of object scene images tobe added with each other into common partial images, and a weightedadder for weighing the division result by said divider with acoefficient for each common partial image and adding the results to eachother.
 30. An image processing apparatus according to claim 29, whereinsaid detector assigns a plurality of detection areas to each of saidplurality of object scene images, and performs a movement detection foreach detection area, said adder further includes a coefficient deciderfor deciding said coefficient for each common partial image, and saidcoefficient decider for specifying detection areas corresponding to eachcommon partial image out of said plurality of detection areas, anddeciding the coefficient on the basis of the movement of the specifieddetection area out of the detection results by said detector.
 31. Animage processing apparatus according to claim 29, wherein said adderfurther includes a difference calculator for calculating for each commonpartial image a difference between the division results by said divider,and a coefficient decider for deciding said coefficient for each commonpartial image, and said coefficient decider performs a coefficientdecision on the basis of the calculation result by said differencecalculator.
 32. An image processing apparatus according to claim 19,wherein said imager includes a CMOS.
 33. An image processing apparatusaccording to claim 19, further comprising said imager.
 34. An imageprocessing apparatus for performing image processing on a plurality ofobject scene images according to a time series that is output from animager for repetitively capturing an optical image of an object scene,comprising: a movement detector for detecting a movement of a featurepoint among said plurality of object scene images; a first clipper forperforming clipping processing on each of said plurality of object sceneimages at a position based on the detection result by said movementdetector; and an adder for performing adding processing of adding toeach of said plurality of object scene images after said first clipper,one or plurality of object scene images temporally proximate to saidobject scene image, wherein said adder includes a divider for dividing apair of object scene images to be added with each other into commonpartial images, and a weighted adder for weighing the division result bysaid divider with a coefficient for each common partial image and addingthe results to each other.
 35. An image processing apparatus accordingto claim 34, wherein said detector assigns a plurality of detectionareas to each of said plurality of object scene images, and performs amovement detection for each detection area, said adder further includesa coefficient decider for deciding said coefficient for each commonpartial image, and said coefficient decider for specifying detectionareas corresponding to each common partial image out of said pluralityof detection areas, and deciding the coefficient on the basis of themovement of the specified detection area out of the detection results bysaid detector.
 36. An image processing apparatus according to claim 34,wherein said adder further includes a difference calculator forcalculating for each common partial image a difference between thedivision results by said divider, and a coefficient decider for decidingsaid coefficient for each common partial image, and said coefficientdecider performs a coefficient decision on the basis of the calculationresult by said difference calculator.
 37. An image processing apparatusaccording to claim 34, further comprising: an FPN corrector forperforming FPN correction processing on said plurality of object sceneimages; a first determiner for repetitively determining whether or notthe movement detected by said movement detector is above a thresholdvalue; and a first controller for making said FPN corrector invalid whenthe determination result by said first determiner is affirmative andmaking said FPN corrector valid when the determination result by saidfirst determiner is negative.
 38. An image processing apparatusaccording to claim 37, further comprising a first memory to which FPNinformation inherent to said imager is written, wherein said FPNcorrector executes FPN correction processing on the basis of the FPNinformation stored in said first memory.
 39. An image processingapparatus according to claim 38, wherein said FPN information includessize information indicating a size of the FPN, and said threshold valueis a value in relation to said size information.
 40. An image processingapparatus according to claim 39, wherein said threshold value is largerthan the value indicated by said size information.
 41. An imageprocessing apparatus according to claim 40, wherein said FPN informationincludes presence or absence information indicating the presence orabsence of an FPN, further comprising: a second memory to which modeinformation indicating whether a camera shake correction mode is validor invalid is written; and an updater for updating the mode informationstored in said second memory in response to a mode switching operation,wherein said first determiner executes determining processing when thepresence or absence information stored in said first memory indicatesthat an FPN is present, and the mode information stored in said secondmemory indicates that the camera shake correction is valid.
 42. An imageprocessing apparatus according to claim 41, wherein said firstcontroller makes said first clipper valid regardless of thedetermination result by said first determiner.
 43. An image processingapparatus according to claim 41, further comprising a second controllerfor making said FPN corrector valid and making said first clipperinvalid when the presence or absence information stored in said firstmemory indicates that an FPN is present and the mode information storedin said second memory indicates that the camera shake correction mode isinvalid.
 44. An image processing apparatus according to claim 43,further comprising a third controller for making said FPN correctorinvalid and making said first clipper valid when the presence or absenceinformation indicates that an FPN is absent and the mode informationstored in said second memory indicates that the camera shake correctionmode is valid.
 45. An image processing apparatus according to claim 44,further comprising a fourth controller for making each of said FPNcorrector and said first clipper invalid when the presence or absenceinformation indicates that an FPN is absent and the mode informationstored in said second memory indicates that the camera shake correctionmode is invalid.
 46. An image processing apparatus according to claim45, further comprising a second clipper for performing clippingprocessing on each of said plurality of object scene images at a fixedposition, wherein each of said first controller and said thirdcontroller further makes said second clipper invalid in response to thevalidating processing of said first clipper, and each of said secondcontroller and said fourth controller further makes said second clippervalid in response to the invalidating processing by said first clipper.47. An image processing apparatus according to claim 34, wherein saidimager includes a CMOS.
 48. An image processing apparatus according toclaim 34, further comprising said imager.